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Introduction 

Kentucky has been a leader in the movement to more rigorous college and career ready standards 
to support their students' success in the 21st century. The first state to adopt new college and 
career ready standards (CCRS), termed the Kentucky Core Academic Standards, Kentucky and 
many of its districts have moved proactively and strategically to meet the challenge of more 
rigorous expectations and to facilitate educators and students' transition to the new demands. 

All students are to be on a trajectory to graduate high school, and should be prepared for college 
and career success. Basic skills have given way to goals for deeper learning, where students 
are expected to apply, reason with, communicate, and use their knowledge to solve complex 
problems. 

This brief summarizes early evidence on the success of two tools Kentucky districts have used to 
support their teachers' transition to these more demanding goals: Literacy Design Collaborative 
(LDC) and Math Design Collaborative (MDC). With support from the Bill and Melinda Gates 
Foundation, LDC and MDC tools have been designed and implemented to embody the key 
shifts in teaching and learning that the new standards demand. By implementing the tools, 
teachers then engage in new pedagogy and address relevant learning goals of the Kentucky Core 
Academic Standards. 

In the sections that follow, we provide a brief background on the two tools and our evaluation 
methodology. We then follow with findings for each intervention and conclude with implications 
of our findings across the two studies. We stress that our methods were rigorous and our findings 
positive, but still our study provides only an "early read" on LDC and MDC effectiveness. Our 
quasi-experimental design cannot separate the effects of LDC and MDC from other changes that 
may have been going on in the study districts and schools. Further, our study is based on a limited 
sample of schools and teachers in select subjects and grade levels who participated in the piloting 
of the tools. These included eighth grade social studies/history and science teachers and ninth 
grade Algebra 1 teachers who initiated their tool use during the 2010-11 or 201 1 -1 2 school years. 
Study results are based on data from the 2012-2013 school year. Full technical reports for each of 
the two studies are also available (Herman et al., 2014 and Herman et al., 201 5). 
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Background 


In this background section, we briefly describe the LDC 
and MDC tools and the evaluation questions our study was 
designed to address. A summary of our research design 
and instrumentation also is provided. 

LDC and MDC Tools 

LDC and MDC tools reflect distinct approaches to 
support teachers’ transition that are consistent with 
the unique challenges in each subject. Even so, both 
interventions reflect a commitment to collaboration in 
design and co-development involving subject matter 
experts, education leaders, and classroom teachers. 


LDC Tool 

LDC provides flexible, module templates that enable 
middle and high school teachers to integrate CCRS 
standards in reading, research, and writing into their 
content area instruction. End-of-module, extended writing 
tasks provide the heart of the approach. Teachers use one 
of many available fill-in-the-blank templates to design 
content-oriented, culminating writing tasks aligned with 
English-language arts (ELA) standards, which are then used 
to organize a module of instruction. After deciding on this 
end-of-module task, teachers then use an LDC-specified 
"instructional ladder" to design activities that will help 
students to develop both content and requisite literacy 
skills to complete the writing task successfully. The steps 
of the ladder include core activities that scaffold student 
learning and provide ongoing opportunities for formative 
assessment. The final product (i.e., the instructional ladder 
and template task) is referred to as a LDC module (see 
http://ldc.org for more information). 

Teachers participating in our LDC study were required 
to implement at least two modules during the academic 
year, with at least one of these targeting explanatory 
writing and the other targeting argument. Teachers 
worked individually or collaboratively with other teachers 
and/or specialists to create the modules, which typically 
underwent a process of review and refinement. 


MDC Tool 

MDC supports the transition by providing one- to two-day 
Classroom Challenges that can be used in conjunction 
with on-going curriculum and instruction. Each Challenge 
essentially is a formative assessment lesson to help 
secondary mathematics teachers develop, assess, and 
monitor their students' development of key mathematical 
skills and understandings. The Challenges also are intended 
to model and help teachers incorporate into their practice 
deeper mathematical reasoning and thinking. Toward 
this end, there are two primary types of mathematical 
challenges: one focusing on conceptual understanding 
and the other on problem solving. Teachers can choose 
from among 40 Challenges aligned towards high school 
standards and 61 Challenges geared towards middle school 
standards to fit with their specific curriculum. 

Developed by the Shell Centre in England in collaboration 
with researchers at UC Berkeley, each Challenge follows the 
same general structure: (a) students engage in a pre-test, 
which presents a challenging problem or question involving 
previously learned concepts or principles, (b) teachers 
review student responses to assess student approaches, 
solution strategies, understandings and misconceptions, (c) 
students are then engaged in whole class and small group 
collaborative activities to discuss alternative approaches, 
surface misconceptions, deepen their understanding, 
and connect and apply their knowledge and skills in new 
contexts, and (d) at the end of the Challenge, students 
return to the initial problem or question, revise their initial 
responses and reflect on what understandings have been 
gained (see http://map.mathshell.org for more information 
about the Classroom Challenges and http://collegeready. 
gatesfoundation.org/Learning/MathDesignCollaborative for 
more information about MDC). 


Teachers in our study typically were required to submit data 
on four to six Challenges over the course of the year, but in 
some cases teachers implemented and submitted data on 
many more. 
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Evaluation Questions 

Parallel evaluation questions guided both studies: 

7. How are teachers implementing the LDC/MDC tools? 

2. What is the impact of LDC/MDC on student learning? 

3. What conditions and contexts influence LDC/MDC 
effectiveness? 


Evaluation Methodology 

Samples 

The LDC study included all 36 eighth grade social studies/ 
history and science teachers and their students in the 1 1 
Kentucky districts who were part of the Phase I and Phase 

11 piloting of LDC, which started in the 2010-1 1 and 201 1- 

12 school years respectively. Half of these teachers agreed 
to provide data for the implementation component of the 
study. 

The implementation component of the MDC study drew 
from a population of 46 ninth grade Algebra I teachers 
determined to be eligible for the study based on their 
participation in Phase I and Phase II piloting, which occurred 
during the same time frame as LDC. Approximately 60% 
of these teachers, representing five districts, provided 
data for the implementation phase of this study. Later 
communication with districts yielded a larger sample of 
59 teachers, which was used in the quasi-experimental 
analyses. 

The majority of both the LDC and MDC samples were 
Phase II teachers, which means that they had only one year 
of implementation experience prior to the study year. 

Implementation Measures 

Multiple measures were used to gauge tool 
implementation, customized to the goals and designs of 
LDC and MDC respectively: 

• Web-based teacher logs: Teachers reported in real 
time the extent to which and how their instruction 
followed the tool structure, their allocation of time to 
various tool components, and their use of formative 
assessment practices. 

• Web-based teacher surveys: At the end of the study 
year, teachers responded to the same issues addressed 
by the logs and also shared their prior experience 
using the tool, extent of professional development, 


leadership support, and collaboration related to tool 
implementation. 

• Artifact analysis: Two LDC modules were collected 
from each participating LDC teacher, and student 
work from 4-6 MDC Challenges was collected from 
each MDC teacher. These were scored as measures of 
implementation quality. 


Student Learning Measures 

Special CRESST measures were used to evaluate student 
performance on LDC and MDC specific goals. The LDC 
version, termed the CRESST Integrated Learning Assessment 
(ILA), was designed to measure both students' literacy 
development relative to the CCRS in ELA and depth of 
content understanding in social studies/history or science. 
Generally mirroring LDC's demands over a two-day period, 
the ILAs first ask students to read several, typical disciplinary 
texts addressing an important topical problem and to 
answer reading comprehension and analysis questions that 
are aligned with the ELA CCRS. 

On day two, students respond to an essay prompt that 
asks them to synthesize what they know with what 
they have read to produce an extended explanation or 
argument related to the given topic; essays are scored 
on multiple dimensions, using a four-point rubric. LDC 
social studies teachers administered an assessment on the 
Reconstruction period in American History, while science 
teachers implemented an assessment on the theory of 
evolution. The CRESST math assessment was designed to 
address the conceptual content of the five most commonly 
used Challenges by study teachers. The assessment 
includes multiple item types, including word problems 
requiring students to justify their reasoning and provide 
evidence-based explanations. The CRESST measures were 
administered in LDC and MDC classrooms only and thus 
provide an indicator of students' current learning, rather 
than evidence of comparative effectiveness of either tool. 

Kentucky state assessment data served this latter purpose 
and were used as the primary indicator of tool effects on 
student learning. The LDC study used students' 

K-PREP English-language arts, writing and social studies 
scores, as well as the prior years' data from KCCT, to judge 
program effects. The MDC study used tenth grade PLAN 
scores for Algebra as the primary outcome. 
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LDC Findings 


LDC Implementation 

Results from teacher logs, surveys, and analysis of teacher- 
created modules and student work provided at least one 
consistent finding: Across all sources, the data showed 
substantial variation across teachers in all aspects of LDC 
implementation, from how teachers allocated instructional 
time across the various components of LDC, to the primary 
organizational forms teachers used in their LDC instruction, 
the reading and writing skills they most emphasized in 
LDC instruction, and the specific strategies they used to 
formatively assess and provide students' feedback on their 
learning. The quality of LDC modules also varied substantially 
across teachers. 

With this variation as a caveat, the findings provide a portrait 
of by whom, how, and with what positive/negative support 
LDC was implemented by the study sample, as well as 
participating teachers' impressions of the tool's effectiveness. 

Who 

Survey results indicate that study teachers were generally 
highly experienced and stable in their positions, having 
spent most of their careers in the same districts and schools. 
Almost all of the study teachers had one to two years of 
experience beyond their initial training in implementing LDC. 
All of the teachers had participated in the development of at 
least one LDC module and the majority had developed two 
or more. Although most of the teachers were required to 
participate in LDC, rather than having volunteered to do so, 
they felt committed as content teachers to help develop their 
students' literacy skills. 

How 

Log, survey, and module analysis results indicate that 
teachers followed the LDC framework. As indicated by 
the logs and surveys, students were largely engaged in 
independent reading and writing during LDC instruction. 
While teachers tended to at least touch upon a wide 
variety of reading and writing skills during this time, they 
spent relatively little time in direct strategy instruction or 
in delivering mini-lessons. Note-taking and summarizing 
appeared to be relatively frequent student activities during 
independent reading, and for social studies classrooms, 
critical reading skills such as citing and evaluating evidence 
and using it to draw conclusions also were in evidence. There 
was little attention paid to critical analysis and synthesis 


skills, such as differentiating fact and opinion, comparing 
arguments, or analyzing authors' perspectives. In writing, 
teachers also reported some attention to a wide range of 
skills, but with a relative emphasis on elements of structure. 
Across both reading and writing, teachers reported 
engaging in frequent formative assessment, involving 
multiple strategies for monitoring student learning, and for 
responding to student misunderstandings as they occurred. 

Analyses of teacher-developed modules provide a window 
into the quality with which LDC was being implemented. 
Fidelity to the LDC framework was judged a relative 
strength in the ratings, and ratings across most of the nine 
dimensions examined approached or achieved moderate 
levels of quality. Results, however, also suggest room for 
improvement, which is to be expected given participating 
content teachers' experience levels with LDC and with 
teaching literacy. 

With What Support 

Survey responses indicated that teachers felt their district 
leadership supported the LDC intervention, but school level 
support was less consistent across the sample. All teachers 
participated in professional development and found it 
beneficial. Teachers found their colleagues collaborative, 
although formal time for planning and collaboration was 
uneven across the sample. Nonetheless, teachers reported 
that collaboration with their peers was very helpful in 
implementing LDC. Science teachers appeared to be less 
involved in professional development and collaboration 
than were their social studies/history peers. 

Attitudes Towards LDC 

Teachers reported that they found LDC a helpful and 
effective tool in meeting a variety of goals, including 
implementing the Kentucky Common Core Standards, 
using formative assessment, incorporating literacy into 
content classrooms, and increasing the rigor of their writing 
assignments. At the same time, although teachers felt 
that LDC had benefited their students writing and college 
readiness, less than half reported that their students had 
"a great deal" of success on their LDC module reading 
and writing mini-tasks or on the final writing task. These 
results suggest that participating teachers may have needed 
additional help with the design and implementation of the 
modules and with their LDC instruction. 
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LDC Effects on Student Learning 


ILA Performance 

The CRESST Integrated Literacy Assessment (ILA), as 
previously described, provides a general barometer on 
student performance on LDC-like reading and writing tasks 
aligned with CCRS, but do not address the question of 
whether LDC improved student learning. 

Descriptive results for both social studies and science 
reveal generally similar performance on their respective 


ILAs. In reading, students scored approximately one-half 
of the total possible points, and in writing their scores 
were roughly 40% of the total possible. Because students' 
essays were scored on five, four-point dimensions, the 
data suggest that students' scores on each dimension 
hovered around two. The scoring rubric defined a Level 3 
as proficient and a score of 2 as basic. 


Assessment 

Component 

n 

Possible 

score 

M 

score 

SD 

Min 

Max 

Social Studies: 

Reading 

252 

18 

9.90 

2.90 

1 

16 

Reconstruction 

Writing 

253 

20 

8.56 

2.89 

5 

19 

Science: 

Reading 

1B6 

15 

7.59 

3.13 

0 

15 

Evolution 

Writing 

132 

20 

7.63 

2.61 

5 

17 


Table l. ILA Descriptive Results 


Effects on K-PREP scores 

The study used advanced statistical design and analysis 
techniques to examine LDC effects on student learning. 
Coarsened exact matching (CEM) was used to create 
a comparison group of students from across Kentucky 
who were nearly identical to study LDC students in 
demographic characteristics, prior achievement, and 
selected school and district characteristics. Hierarchical 
linear modeling (HLM) was then used to test LDC's effect 
by comparing students' K-PREP performance in reading, 
writing, and social studies to that of the comparison 
group. The analysis also took account of both school 
and teacher effectiveness prior to LDC's implementation, 
based on state assessment data from 2009-2010, and 
controlled for all student background characteristics. The 
models also examined potential interactions between the 
LDC treatment, prior school and teacher effectiveness as 
well as student characteristics. Lor this study, a statistically 
significant interaction effect indicated that the treatment 
had differential effects on student learning depending on 
the school, teacher, and/or individual student's standing 
on the given variable. These interaction analyses should 
be considered highly exploratory and results treated as 
tentative. 


Level 2 variables 

Model coefficient 
(S.E.) 

LDC treatment 

0.058 (0.023)* 

Level 1 
treatment 
by student 
characteristic 
interactions 

Gender 

-0.004(0.017) 

Sped 

-0.110(0.034)* 

FreeLunch 

0.053(0.017)* 

PriorAchiev 

0.034(0.011)* 


Fixed effects for demographic predictors and for prior school and 
teacher effectiveness not shown; * p < .05. 


Table 2. 2012-13 LDC Student Effect Estimates on 
K-PREP Reading, Including Interactions with 
Prior Teacher Effectiveness and 
Student Characteristics 
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Table 2 (previous page) shows HLM results for the K-PREP 
reading scores. The data indicate that LDC had a small 
but statistically significant, positive effect on students' 
reading performance. LDC students scored higher in 
reading than did their carefully matched comparison 
group, demonstrating that LDC had a measurable effect on 
students' literacy learning. 

To provide a benchmark for interpreting this effect, we 
used a relatively new methodology to convert the effect 
size into a gross indicator of the number of months of 
learning represented (see Hill, Bloom, Black, & Lipsey, 
2007). Relative to typical growth in reading from eighth to 
ninth grade, the effect size for LDC represents 2.2 months 
of schooling. 

The data also shows interactions between LDC effects and 
student characteristics. Both students' prior achievement, 
based on their prior year K-PREP scores, and students' 
socio-economic status (SES), as revealed by their free or 


Level 2 variable 

Model coefficient 
5 (S.E.) 

LDC treatment 

-0.026 (0.023) 

LDC treatment by 
teacher effectiveness 

-0.288 (0.082)* 

LDC treatment 
by student 
characteristic 
interactions 

Gender 

0.013(0.016) 

Sped 

-0.007 (0.037) 

FreeLunch 

0.039(0.019)* 

PriorAchiev 

0.050(0.017)* 


Fixed effects for demographic predictors and for prior school and 
teacher effectiveness not shown; * p s .05 


Table 3. 2012-13 LDC Student Effect Estimates on 
Social Studies K-PREP Scores, Including 
Interaction with Teacher Effectiveness 


reduced price lunch status, show positive interactions 
with the treatment. That is, LDC students who were 
relatively higher achieving prior to their LDC experience 
showed relatively greater benefit than did those who 
started relatively lower achieving, although the observed 
effect is very small. Interestingly, LDC students receiving 
free or reduced price lunch, who tended to be relatively 
lower performing, also appeared to have benefited more 
from LDC. Although, again, the observed effect was very 
small, we speculate that struggling LDC students perhaps 
had access to special resources (e.g., Title 1 programs, 
specialist teachers) that provided essential support. We did 
not find evidence of differential effects of LDC by gender. 
Controlling for other factors, special education students 
appear to do less well under LDC; however, the share of 
students falling into this category was small. 

The results for the K-PREP social studies results are shown 
in Table 3. The coefficient for the main effect for LDC 
is small and not statistically significant, indicating that 
LDC's addition of literacy to course requirements did not 
diminish students' content performance. Table 2 also 
reveals a significant interaction between prior teacher 
effectiveness and LDC. LDC students taught by teachers 
who were relatively less effective prior to LDC benefited 
relatively more than did students of relatively more 
effective teachers. However, this interaction is difficult to 
interpret and should be treated cautiously given that all 
teachers', including science teachers, prior effectiveness 
scores were based on their eighth grade students' social 
studies performance for the study's baseline year (because 
Kentucky does not assess science in eighth grade). 

Students' prior year performance on the K-PREP and their 
free or reduced price lunch status show the same, small 
positive interaction with LDC. LDC students who started 
the year relatively higher performing experienced more 
benefit from LDC in their social studies performance, as 
did students who were from a relatively lower SES, as 
evidenced by their free or reduced price lunch status. We 
did not find differential treatment effects of LDC by gender 
or special education status. 

It should also be noted that the results for K-PREP writing 
showed neither a main nor an interaction effects for LDC 
and thus are not presented here (for more detail, see 
Herman et al., 2014). 
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MDC Results 


MDC Implementation 

Results from teacher logs, surveys and analysis of student 
work from MDC's Classroom Challenges, indicate that 
teachers adhered to the MDC challenge design and followed 
expected guidelines. At the same time, however, like the 
LDC results, the data from all sources suggest that teachers 
differed in their specific implementation strategies. This 
included how much time teachers spent on the pre-test to 
how they allocated instructional time across the Challenges' 
various components, composed peer groups, responded 
to struggling students, and the strategies they used to 
formatively assess and provide students' feedback on their 
learning. Analysis of student work on the Challenges also 
showed variation in teachers' success. 

With this variation as a caveat, the findings provide a 
portrait of by whom, how, and with what support MDC was 
implemented by the study sample, as well as participating 
teachers' impressions of the tool's effectiveness. 

Who 

Study teachers showed a range of teaching experience, with 
an average of about nine years. Most had spent the bulk of 
their careers in their current districts and schools. Nearly two- 
thirds indicated that they had volunteered, rather than being 
required, to work with the MDC tool. Most teachers, as 
noted earlier, had only one year of experience implementing 
MDC beyond their initial training year. On average, study 
teachers implemented six Challenges during the study year 
and five during the prior year. 

How 

Nearly all teachers reported that they completely or mostly 
adhered to the MDC guidelines in implementing their 
Challenges, which typically took two to three days for them 
to complete. Teachers, however, varied considerably in the 
time they spent analyzing students' pre-tests and in the 
time and specific strategies they used in implementing each 
component (i.e., the pre-test, collaborative small group 
activity, whole class instruction, and post-test). Although 
teachers tended to deviate from guidelines in the feedback 
they provided to students about their pre-test performance, 
their reported interactions during small group collaborative 
activities and whole class instruction were in sync with 


MDC's productive struggle philosophy. Teachers raised 
questions, asked students to explain their reasoning, 
solicited feedback from peers and asked students to 
self reflect, rather than providing students directly with 
answers. 

Analyses of student performance on the Classrooms 
Challenges, as judged by evidence of student learning 
from pre- to post-test, however, shows a somewhat 
different picture of implementation fidelity. In general, 
these analyses showed that teachers had difficulty helping 
students to achieve the Challenge goals. Results indicate 
that improvements in student understanding were 
generally scant and evidence of misconceptions remained. 
Students appeared to particularly struggle with mounting 
explanations of their reasoning. 

With What Support 

Survey responses indicated that teachers felt strong support 
for MDC from district leadership, but school level principal 
support was less consistent across the sample. The majority 
(75%) reported participating in professional development 
to prepare for MDC implementation and all who did 
found it beneficial. Although formal time for collaborative 
planning and feedback varied across the sample, the great 
majority of teachers reported meeting informally at least 
every other week to discuss their MDC work. Teachers 
also reported that they found their peers to be highly 
collaborative and the collaboration helpful to their MDC 
implementation. 

Attitudes Towards MDC 

Teachers were very positive about key MDC pedagogical 
strategies such as teacher as facilitator, asking guiding 
questions and peer-to-peer problem solving, and also 
found the small group and plenary approaches helpful 
to student learning. Although they felt that that the 
Challenges benefited students conceptual understanding 
and mathematical thinking, they reported that sizeable 
proportions of their students struggled during the 
Challenges and failed to achieve a firm grasp of the 
intended target-on average they felt that only about half of 
their students reached this level. As with the LDC findings, 
these results suggest that participating teachers may have 
needed additional help with the implementation of the 
modules and with their MDC instruction. 
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MDC Effects on Student Learning 


CRESST Math Measure 

The CRESST measure, as previously noted, was specifically 
designed to align with the mathematics concepts addressed 
in the five Challenges that were most commonly used 
by study MDC teachers. The assessment also addresses 
students' ability to justify their reasoning, explain major 
concepts, and practices highly valued in CCRS. Student 
performance on the measure serves as a barometer of 
the status of MDC students learning, but because the 
assessment was administered only in MDC classrooms, it 
could not be used to examine MDC effectiveness. 


Descriptive results on student performance are shown in 
Table 4. On the extended response items, in which students 
were called upon to justify their solutions to word problems 
and explain mathematical concepts, students achieved 
only about one-quarter of the total possible points. Their 
performance was only slightly better on the multiple choice 
and short answer item types. We return to this issue in 
the conclusion, but the data suggest the distance students 
remain from the expectations of CCRS. 


Type 

n 

Possible 

score 

M 

score 

SD 

Min 

Max 

Extended response 

471 

11 

2.83 

2.26 

0 

10 

Multiple choice and short answer 

471 

22 

6.71 

3.62 

0 

18 

Overall 

471 

33 

9.55 

5.39 

0 

28 


Table 4. Descriptive Statistics for CRESST Math Assessment 


Effects on PLAN algebra scores 

As with the LDC study, our MDC analyses used 
sophisticated CEM techniques to create a comparison 
group of students from across Kentucky who were 
nearly identical to the MDC students in demographic 
characteristics, prior achievement, and selected school and 
district characteristics. HLM was used to test MDC's effect 
by comparing students' PLAN performance in algebra 
to that of the comparison group. The analyses also took 
account of both school and teacher effectiveness prior 
to MDC's implementation, based on state assessment 
data from 2009-2010, and controlled for all student 
background characteristics. The models also examined 
potential interactions between the MDC treatment, 
prior teacher and school effectiveness, and student 
characteristics. A statistically significant interaction effect 
indicates that MDC had differential effects on students' 
learning, depending upon their school and/or teacher's 
prior effectiveness and/or their individual demographic or 
prior achievement. These interaction analyses should be 
considered exploratory and any results should be treated 


Level 2 variables 

Model coefficient 
(S.E.) 

MDC treatment 

0.130(0.030)** 

MDC treatment by teacher 
effectiveness 

0.420(0.178)** 


Female 

0.005(0.026) 

MDC treatment 
by student 
characteristics 
interaction 

Special 

education 

0.070(0.044) 

Free/reduced 
price lunch 

0.027 (0.039) 


Prior 

Achievement 

0.030(0.016) 


Fixed effects for demographic predictors and for prior school and teacher 
effectiveness not shown. * p < .05. ** p < .01 . 


Table 5. 2012-13 MDC Student Effect Estimates 
on PLAN Algebra, Including Interaction with 
School and Teacher Effectiveness 
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MDC Effects on Student Learning (cont.) 


as tentative. Two different HLMs using slightly different 
approaches to modeling students' exposure to LDC 
teaching were estimated, and results were very similar 
across the two models. 

Table 5 (previous page) reveals both main and 
interaction effects for the MDC treatment for one 
of the two models. The treatment shows a small, 
statistically significant, positive effect on students' PLAN 
performance in algebra, indicating that students who 
experienced MDC learned more than students who did 
not have the benefit of MDC. As with LDC, we used 
recently developed methodology to convert the observed 
effect size for MDC into a gross indicator of the number 
of months of learning represented (see Hill et al., 2007). 
Relative to typical growth in mathematics from ninth 
to tenth grade, the effect size for MDC represents 4.6 
months of schooling. 

The results also show an interaction effect for prior 
teacher effectiveness, but in the opposite direction to 
that found for LDC. Here we find a positive interaction 
effect with MDC, meaning that students whose teachers 
were more effective prior to using MDC, as measured 


by their students' (pre-MDC) state assessment scores 
in mathematics, benefited more from MDC compared 
to students with lower "value-added" teachers. 

We speculate that the more effective teachers had 
higher pedagogical knowledge in mathematics (see, 
for example, Hill et al., 2007) that enabled them to 
implement MDC better. The interaction between 
treatment and student prior achievement was close to 
being statistically significant at the 5 percent level in 
the model shown. In the alternate model, the statistical 
significance of the interaction effects were reversed, 
with the student prior achievement interaction being 
statistically significant and the prior teacher effectiveness 
interaction approaching significance (See Herman et al., 
2015 for additional detail). Given the likely relationship 
between teacher quality and student achievement, 
these inconsistencies are understandable. The combined 
findings suggest MDC students who were relatively 
higher achieving prior to their MDC experience and/or 
those whose teachers were more effective prior to using 
MDC benefited more from the intervention compared to 
their peers. 
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Implications 


LDC and MDC are very different tools but our Kentucky 
studies of them reveal a strong similarity in at least one 
respect: They both show promising, positive results in 
supporting teachers' transition to the Kentucky Core 
Academic Standards expectations for college and career 
readiness and in improving student learning. At the same 
time, however, study findings suggest challenges that LDC 
and MDC will need to overcome to move to higher levels 
of success. 

Positive Effects on Student Learning 

That LDC and MDC show statistically significant results 
on students' state assessment scores in reading and 
mathematics respectively are results worth celebrating. 
These positive findings are particularly so in light of both 
study teachers' limited prior experience implementing the 
tools and the limited dosage students experienced. That 
is, study teachers had only one or two years of experience 
with LDC or MDC prior to the study year, and for the 
great majority it was only one year. Based on research 
on teachers' implementation of new practices, this is 
insufficient time for teachers to become fully comfortable 
and competent with the kinds of new pedagogical 
practices that LDC and MDC represent (Coburn, 2003; 
Hargreaves & Fullan, 2012). Consider that LDC requires 
that content teachers teach literacy, a new responsibility 
for which they have had little or no prior training. MDC, 
for its part, emphasizes student-centered pedagogies 
and the engagement of students' mathematical thinking 
and problem solving learning, in contrast to the teacher- 
directed approaches and low-level learning goals that have 
been dominant in traditional mathematics curriculum (U.S. 
Department of Education, 2008). 

Intervention dosage is another factor to consider in 
evaluating LDC and MDC effects. In general, the longer 
and more intensive the treatment, the more likely an 
intervention is to show measurable effects. LDC teachers 
typically implemented two modules of two-to-three weeks' 
duration each during the study year, meaning that LDC- 
oriented coursework totaled four-to-six weeks, only a small 
fraction of the full academic year. For MDC, participating 
teachers were expected to implement between four and 


six Challenges, meaning that students were engaged only 
8-12 days of the school year. 

Nonetheless, the studies found statistically significant 
learning effects for both tools, the approximate 
equivalent of 2.2 months of schooling for LDC and 
4.6 months or MDC. Given their contexts of early 
implementation and limited dosage, these small effects 
are noteworthy. 

Positive Effects on Teachers 

The effects found for student learning are matched by 
teacher enthusiasm for the two tools. Across LDC and 
MDC, teachers were positive about the professional 
development they received and reported that they 
found the tools helpful and effective in meeting a 
variety of goals, including implementing Kentucky 
Core Academic Standards, using formative asessment, 
incorporting more complex thinking and problem solving 
into curriculum and instruction, and improving student 
learning. Teachers' reports about their fidelity of tool 
implementation provided additional evidence of their 
positive attitudes. 

Struggles in 

Moving to Higher Standards 

While our study found positive effects on teachers and 
students, findings also demonstrated the challenge of 
moving to Kentucky's more rigorous Core Academic 
Standards. We see evidence of this challenge in students' 
low performance on measures specifically designed to 
reflect the deeper learning demands of new college 
and career ready standards and in teachers' reports that 
sizeable proportions of their students are struggling 
relative to the goals of LDC and MDC. Our analysis 
of LDC and MDC classroom artifacts also indicate 
that some teachers struggled in their implementation 
efforts, as would be expected given this early stage of 
implementation. 

That some teachers and students struggled is not meant 
to imply that current standards are unattainable or that 
college and career ready expectations for students should 
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be reduced — after all, we know that returning to prior 
standards will not get our children to 21st century success. 
However, the evidence does suggest that change will not 
come overnight and that both teachers and students will 
need support to meet the challenge. The issue is two-fold: 
(a) how to address the needs and better prepare students 
and teachers who may not yet be ready to be successful 
with the challenges of LDC and MDC; and (b) how to 
modify and/or adapt the tools to scaffold teacher and 
student learning more effectively. 


Achievement Gap Implications 

Although we regard findings of the interaction between 
student characteristics and treatment effects as tentative 
and subject to further validation, the consistency in results 
across measures and across LDC and MDC is striking. 

While the overall results indicated that both tools were 
effective for all students, the interaction findings suggested 
that initially higher achieving students benefited more than 
did initially lower achieving students. Such a finding makes 
intuitive sense in that lower achieving students have most 
likely been exposed to the "drill and kill" test preparation 
curriculum of the past, are least likely to have acquired the 
prior grade knowledge and skills expected by the Kentucky 
Core, and are least likely to have been engaged in the 
deeper conceptual understanding and applications that 
mark the new standards. 

However understandable, the findings thus suggest that, 
in the absence of additional scaffolding and supports for 
low achieving students, LDC and MDC are likely to be gap 
enhancing. Study findings on the interaction between prior 
teacher effectiveness and treatment reveal a similar story — 
students of relatively more effective teachers appeared 
to benefit more from MDC than did students of relatively 
less effective teachers. To the extent that low achieving 
students are more likely to be assigned to relatively 
less effective teachers, these students will be doubly 
disadvantaged. 

Study findings of a positive interaction between LDC 
and students' free and reduced price lunch status offers 
promise for future inquiry. The results suggest that, 
controlling for prior achievement and other background 
characteristics, students who are more economically 
disadvantaged fare relatively better under LDC than their 
more advantaged peers. These results were not found for 
MDC. 


Strengthening Implementation 

Although teachers reported implementing all components 
of both LDC and MDC, the findings suggest substantial 
variation in how they implemented each tool and in the 
relative time and specific strategies they used in doing 
so. The study did not achieve strong findings with regard 
to what aspects of implementation mattered most or of 
what specific strategies were most effective. The findings 
are suggestive, however, of some factors that might 
be important to success: District support for the two 
interventions was clear across the sample, yet principal 
or local school support was more variable, suggesting a 
potential problem point. Teachers found their peers highly 
collaborative and helpful in implementing the two tools. 
The extent of such collaboration tended to be associated 
with MDC success. Moreover, having a teacher who 
volunteered to participate in MDC, rather than having 
participation required, was associated with MDC effects on 
student learning. 
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Conclusions 


In summary, our studies reveal that study teachers are 
enthusiastic about both LDC and MDC, and that both 
showed important effects on student learning. Even 
so, study results also suggest areas for improvement. 
Content teachers who implement LDC likely will be more 
successful to the extent they have expertise in supporting 
students' literacy development. Mathematics teachers who 
implement MDC likely will be more successful to the extent 
they possess sufficient pedagogical-content knowledge 
to effectively scaffold students productive struggle with 
complex mathematical concepts and problems. Struggling 
students may need additional supports and strategies to 
acquire the prerequisite knowledge and skills assumed 


by MDC's Classroom Challenges to move their productive 
struggles to success in new mathematical understandings 
and problem solving. 

We leave it to future research to examine the 
generalizability of these findings in the larger samples of 
teachers and schools that are now implementing LDC and 
MDC. Cost effectiveness studies also should be of interest. 
Future research and development also should continue 
the quest to identify both the most critical aspects of 
implementation in improving student learning and key 
infrastructure and supports that students and teachers who 
currently are struggling need to propel their success. 
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